Assignment #5 - Deep Learning (Extra Credits)

DUE: Dec 6 (Friday) 11:00 pm

Amit Shetty

I. Overview

Describe the objective of this assignment. You can briefly state how you accompilsh it.

The objective of this assignment is to understand what deep learning is. Deep learning is a branch of machine learning that forms the basis of artificial neural networks. To understand deep learning, I will be focussing on designing different types of neural neural networks. The process I will be following will be to collect the data and visualisae it to gain some key insights. This will allow me to decide what my input and target variables must be. I will be using keras (https://keras.io/) to implement the neural networks. Keras creates neural networks in the form of models which I will them apply to the data to get some predictons.

II. Data

Introduce your data and visualize them. Describe your observations about the data. You can reuse the data that you examined in Assignment #0 (of course for classification).

The two datasets are related to red and white variants of the Portuguese "Vinho Verde" wine. Due to privacy and logistic issues, only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.).

These datasets can be viewed as classification or regression tasks. The classes are ordered and not balanced (e.g. there are many more normal wines than excellent or poor ones).

While the datasets are divied into two different wine types, the characters for calulcating the quality of wine still remains the same for both wines

Input variables (based on physicochemical tests): 1 - fixed acidity 2 - volatile acidity 3 - citric acid 4 - residual sugar 5 - chlorides 6 - free sulfur dioxide 7 - total sulfur dioxide 8 - density 9 - pH 10 - sulphates 11 - alcohol

Target variable (based on sensory data): 12 - quality (score between 0 and 10)

In [1]:
import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import keras
from keras.models import Sequential
from keras.layers import Dense, LSTM, GlobalMaxPooling1D, SpatialDropout1D, Embedding, Dropout
from sklearn.preprocessing import OneHotEncoder
from sklearn.model_selection import train_test_split
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.metrics import accuracy_score
import tensorflow
import time
%matplotlib inline
Using TensorFlow backend.
In [2]:
df_red = pd.read_csv("winequality_red.csv")
df_white = pd.read_csv("winequality_white.csv")

We will be adding a new column to our red and white wine datasets tom add a new color column to be able to differentiate between the wines when the datasets are merged

In [3]:
df_red["color"] = "R"
In [4]:
df_white["color"] = "W"

Merging the two red and white wine datasets together for eventual training and testing

In [5]:
df_all=pd.concat([df_red,df_white],axis=0)
In [6]:
df_all.head()
Out[6]:
fixed acidity volatile acidity citric acid residual sugar chlorides free sulfur dioxide total sulfur dioxide density pH sulphates alcohol quality color
0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 R
1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.9968 3.20 0.68 9.8 5 R
2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.9970 3.26 0.65 9.8 5 R
3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.9980 3.16 0.58 9.8 6 R
4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 R

To avoid any issues withspaces when processing data, we will be renaming the data columns and replacing the spaces with _ characters

In [7]:
df_white.rename(columns={'fixed acidity': 'fixed_acidity','citric acid':'citric_acid','volatile acidity':'volatile_acidity','residual sugar':'residual_sugar','free sulfur dioxide':'free_sulfur_dioxide','total sulfur dioxide':'total_sulfur_dioxide'}, inplace=True)
In [8]:
df_red.rename(columns={'fixed acidity': 'fixed_acidity','citric acid':'citric_acid','volatile acidity':'volatile_acidity','residual sugar':'residual_sugar','free sulfur dioxide':'free_sulfur_dioxide','total sulfur dioxide':'total_sulfur_dioxide'}, inplace=True)
In [9]:
df_all.rename(columns={'fixed acidity': 'fixed_acidity','citric acid':'citric_acid','volatile acidity':'volatile_acidity','residual sugar':'residual_sugar','free sulfur dioxide':'free_sulfur_dioxide','total sulfur dioxide':'total_sulfur_dioxide'}, inplace=True)
In [10]:
df_all.head()
Out[10]:
fixed_acidity volatile_acidity citric_acid residual_sugar chlorides free_sulfur_dioxide total_sulfur_dioxide density pH sulphates alcohol quality color
0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 R
1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.9968 3.20 0.68 9.8 5 R
2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.9970 3.26 0.65 9.8 5 R
3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.9980 3.16 0.58 9.8 6 R
4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5 R

Dummy variables for modelling. other variables will be normalised when the models are created

In [11]:
df = pd.get_dummies(df_all, columns=["color"])
In [12]:
df
Out[12]:
fixed_acidity volatile_acidity citric_acid residual_sugar chlorides free_sulfur_dioxide total_sulfur_dioxide density pH sulphates alcohol quality color_R color_W
0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4 5 1 0
1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.99680 3.20 0.68 9.8 5 1 0
2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.99700 3.26 0.65 9.8 5 1 0
3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.99800 3.16 0.58 9.8 6 1 0
4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4 5 1 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
4893 6.2 0.21 0.29 1.6 0.039 24.0 92.0 0.99114 3.27 0.50 11.2 6 0 1
4894 6.6 0.32 0.36 8.0 0.047 57.0 168.0 0.99490 3.15 0.46 9.6 5 0 1
4895 6.5 0.24 0.19 1.2 0.041 30.0 111.0 0.99254 2.99 0.46 9.4 6 0 1
4896 5.5 0.29 0.30 1.1 0.022 20.0 110.0 0.98869 3.34 0.38 12.8 7 0 1
4897 6.0 0.21 0.38 0.8 0.020 22.0 98.0 0.98941 3.26 0.32 11.8 6 0 1

6497 rows × 14 columns

In [13]:
# Checking for nay null values
df_all.isnull().sum()
Out[13]:
fixed_acidity           0
volatile_acidity        0
citric_acid             0
residual_sugar          0
chlorides               0
free_sulfur_dioxide     0
total_sulfur_dioxide    0
density                 0
pH                      0
sulphates               0
alcohol                 0
quality                 0
color                   0
dtype: int64
In [14]:
df_all.describe()
Out[14]:
fixed_acidity volatile_acidity citric_acid residual_sugar chlorides free_sulfur_dioxide total_sulfur_dioxide density pH sulphates alcohol quality
count 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000 6497.000000
mean 7.215307 0.339666 0.318633 5.443235 0.056034 30.525319 115.744574 0.994697 3.218501 0.531268 10.491801 5.818378
std 1.296434 0.164636 0.145318 4.757804 0.035034 17.749400 56.521855 0.002999 0.160787 0.148806 1.192712 0.873255
min 3.800000 0.080000 0.000000 0.600000 0.009000 1.000000 6.000000 0.987110 2.720000 0.220000 8.000000 3.000000
25% 6.400000 0.230000 0.250000 1.800000 0.038000 17.000000 77.000000 0.992340 3.110000 0.430000 9.500000 5.000000
50% 7.000000 0.290000 0.310000 3.000000 0.047000 29.000000 118.000000 0.994890 3.210000 0.510000 10.300000 6.000000
75% 7.700000 0.400000 0.390000 8.100000 0.065000 41.000000 156.000000 0.996990 3.320000 0.600000 11.300000 6.000000
max 15.900000 1.580000 1.660000 65.800000 0.611000 289.000000 440.000000 1.038980 4.010000 2.000000 14.900000 9.000000

Plotting correlation matrices for both red and white wines individually and for the whole combination

In [15]:
plt.subplots(figsize=(20,15))
ax = plt.axes()
ax.set_title("Wine Characteristic Correlation Heatmap (Reds)")
corr = df_red.corr()
sns.heatmap(corr, 
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values, annot=True, cmap="Reds")
plt.show()
In [16]:
plt.subplots(figsize=(20,15))
ax = plt.axes()
ax.set_title("Wine Characteristic Correlation Heatmap (Reds)")
corr = df_white.corr()
sns.heatmap(corr, 
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values, annot=True, cmap = "Blues")
plt.show()
In [17]:
plt.subplots(figsize=(20,15))
ax = plt.axes()
ax.set_title("Wine Characteristic Correlation Heatmap (All wines)")
corr = df_all.corr()
sns.heatmap(corr, 
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values, annot=True, cmap="Oranges")
plt.show()

Testing association between density of wine and sugar content of red, white and combined wine dataset

In [18]:
scat1 = sns.regplot(x = "density", y = "residual_sugar", fit_reg = True, color='r', data = df_red)
plt.xlabel("Density of wine")
plt.ylabel("Residual sugar in wine in grams")
plt.title("Association between RED wine's density and residual sugar")
plt.show()
In [19]:
scat1 = sns.regplot(x = "density", y = "residual_sugar", fit_reg = True, color='b', data = df_white)
plt.xlabel("Density of wine")
plt.ylabel("Residual sugar in wine in grams")
plt.title("Association between WHITE wine's density and residual sugar")
plt.show()

We will now be checking how the quality of wine is distributed for red and white wines and combined dataset

In [20]:
df_red["quality"] = pd.Categorical(df_red["quality"])
sns.countplot(x="quality", data=df_red)
plt.xlabel("Quality level of RED wine (0-10 scale)")
plt.show()
In [21]:
df_white["quality"] = pd.Categorical(df_white["quality"])
sns.countplot(x="quality", data=df_white)
plt.xlabel("Quality level of WHITE wine (0-10 scale)")
plt.show()

One of the key factors affecting the quality is the amount of alchohol in the wine

In [22]:
sns.factorplot(x="quality", y="alcohol", data=df_red, kind="strip")
plt.xlabel("Quality level of wine, 0-10 scale")
plt.ylabel("Alcohol level in wine in % ABV")
plt.title("Alcohol percent in each level of RED wine's quality")
plt.show()
C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\seaborn\categorical.py:3666: UserWarning: The `factorplot` function has been renamed to `catplot`. The original name will be removed in a future release. Please update your code. Note that the default `kind` in `factorplot` (`'point'`) has changed `'strip'` in `catplot`.
  warnings.warn(msg)
In [23]:
sns.factorplot(x="quality", y="alcohol", data=df_white, kind="strip")
plt.xlabel("Quality level of wine, 0-10 scale")
plt.ylabel("Alcohol level in wine in % ABV")
plt.title("Alcohol percent in each level of WHITE wine's quality")
plt.show()

We will be seeing the distribution of volatile acidity over both red and white wine

In [24]:
redlabels = np.unique(df_red['quality'])
whitelabels = np.unique(df_white['quality'])
fig, ax = plt.subplots(1, 2, figsize=(12, 6))
plt.title('Distribution of Voltile Acidity in Red and White Wine')
redcolors = np.random.rand(6,4)
whitecolors = np.append(redcolors, np.random.rand(1,4), axis=0)

for i in range(len(redcolors)):
    redy = df_red['alcohol'][df_red.quality == redlabels[i]]
    redx = df_red['volatile_acidity'][df_red.quality == redlabels[i]]
    ax[0].scatter(redx, redy, c=redcolors[i])
for i in range(len(whitecolors)):
    whitey = df_white['alcohol'][df_white.quality == whitelabels[i]]
    whitex = df_white['volatile_acidity'][df_white.quality == whitelabels[i]]
    ax[1].scatter(whitex, whitey, c=whitecolors[i])
    
ax[0].set_title("Red Wine")
ax[1].set_title("White Wine")
ax[0].set_xlim([0,1.7])
ax[1].set_xlim([0,1.7])
ax[0].set_ylim([5,15.5])
ax[1].set_ylim([5,15.5])
ax[0].set_xlabel("Volatile Acidity")
ax[0].set_ylabel("Alcohol")
ax[1].set_xlabel("Volatile Acidity")
ax[1].set_ylabel("Alcohol") 
ax[1].legend(whitelabels, loc='best', bbox_to_anchor=(1.3, 1))

plt.show()
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.
'c' argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with 'x' & 'y'.  Please use a 2-D array with a single row if you really want to specify the same RGB or RGBA value for all points.

To summarise the data collection, plotting the correlation between all the different variables with the final quality

In [25]:
sns.pairplot(df_red, vars=df_red.columns[:-1])
plt.title("Pair plot showing RED wine charcteristics")
plt.show()
In [26]:
sns.pairplot(df_white, vars=df_white.columns[:-1])
plt.title("Pair plot showing WHITE wine charcteristics")
plt.show()
In [27]:
sns.pairplot(df_all, vars=df_all.columns[:-1])
plt.title("Pair plot showing ALL wine charcteristics")
plt.show()

Results Summary Discussion

  • We have two target variables in our dataset. Each serves different purposes. The 'quality' column determines form a scale from 0-10 which will be used in our regression modelling. The 'color' column has 2 unique categories 'Red' and 'White' whihc is the type of wine which meets the characteristics

  • The input variables will be the attributes that are determined by people's subjectives observations and certain known qualities of the wine.

  • From the correlation matrix we can see that citric acid, sulphates, fixed acidity and alcholhol content determine how good the red and white wine is.

  • The denisty ditribution shows that red wine is more dense than white wine which will have an obvious effect on its taste and corresponding attributes.

  • However despite the previous observation of red wine being richer than white wine, the quality distribution shows that white wine enjoys a more priveleged position

  • The scatter plot show variation of alchohol levels in white wine and red wine where red wine has a more scattered densoty distribution compared to white wine.

Refer: https://www.datacamp.com/community/tutorials/deep-learning-python

III. Methods

In this assignment, you are building a deep network with more than 5 layers using TensorFlow. Looking at the chart below, get some idea about how you can construct your networks for what problem and why you pick your structure.

  • Pick at least 3 different networks for experiments.
  • Summarize the choice of your networks.
  • Explain your TensorFlow (or Keras) codes.
  • Explain how you apply your model to your data.

Following images are only for you to get some idea. You do not necessarily stick with these. You can come up with your own structure or shape.

We now have to determine the input and the target variables will be. In our case, target variable will be the type of wine i.e. red or white depending on the remaining attributes in our dataset.

In [28]:
X = df_all.iloc[:, :11]
X.shape
Out[28]:
(6497, 11)
In [29]:
T = df_all.iloc[:, 12:13]
T.shape
Out[29]:
(6497, 1)
In [30]:
# Apply one hot encoder
ohe = OneHotEncoder()
T = ohe.fit_transform(T).toarray()
In [31]:
X_train,X_test,T_train,T_test = train_test_split(X,T,test_size = 0.2)
In [32]:
T
Out[32]:
array([[1., 0.],
       [1., 0.],
       [1., 0.],
       ...,
       [0., 1.],
       [0., 1.],
       [0., 1.]])

Method 1: Simple Feed Forward Neural Network Model (Classification)

The simplest kind of neural network is a single-layer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called artificial neurons or linear threshold units. The goal of a feedforward network is to approximate some function f. For example, for a classifier, y = f(x) maps an input x to a category y. A feedforward network defines a mapping y = f(x;θ) and learns the value of the parameters θ that result in the best function approximation. These models are called feedforward because information flows through the function being evaluated from x, through the intermediate computations used to define f, and finally to the output y. There are no feedback connections in which outputs of the model are fed back into itself.

Implementation Technique

  • I will be using Keras to build the neural network model as it makes it easier to easily tweak the paramters needed to build each layer of the neural network

  • As I am building a basic neual network for my initial of the 3 models to build, I will be vuilding using a one input layer, two hidden layers and one output layer with 2 neurons which is the scope of the data at hand.

  • The input dimensions will be noted at 11 whihc is the attibutes that I am using as the deciding factor for the red and white wine classification.

  • The activation function used for the hidden layers is rectified linear units (ReLU) and the final activation function used for classifiction will be the softmanx function.

  • Keras provides an option to compile the model using epochs and batch sizes for muliple loads to be send for compilation.

  • We will also be checking how the model performs by issuing the test data as a validation set and compiling the model.

In [33]:
model = Sequential()
model.add(Dense(20, input_dim=11, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(2, activation='softmax'))
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

In [34]:
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

In [35]:
# history = model.fit(X_train, T_train, epochs=100, batch_size=64)
In [36]:
# T_pred = model.predict(X_test)
# #Converting predictions to label
# pred = list()
# for i in range(len(T_pred)):
#     pred.append(np.argmax(T_pred[i]))
# #Converting one hot encoded test label to label
# test = list()
# for i in range(len(T_test)):
#     test.append(np.argmax(T_test[i]))
In [37]:
# pred
In [38]:
# test

Applyting model by training and validating using test set

In [39]:
start_time_1 = time.time()
history = model.fit(X_train, T_train,validation_data = (X_test,T_test), epochs=100, batch_size=64)
end_time_1 = time.time()
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\tensorflow_core\python\ops\math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:2741: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Train on 5197 samples, validate on 1300 samples
Epoch 1/100
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

5197/5197 [==============================] - 1s 115us/step - loss: 8.4306 - acc: 0.4099 - val_loss: 0.3108 - val_acc: 0.8769
Epoch 2/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.2654 - acc: 0.9009 - val_loss: 0.2317 - val_acc: 0.9162
Epoch 3/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.2301 - acc: 0.9142 - val_loss: 0.2195 - val_acc: 0.9185
Epoch 4/100
5197/5197 [==============================] - 0s 15us/step - loss: 0.2176 - acc: 0.9217 - val_loss: 0.2076 - val_acc: 0.9231
Epoch 5/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.2027 - acc: 0.9271 - val_loss: 0.1988 - val_acc: 0.9285
Epoch 6/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.1996 - acc: 0.9292 - val_loss: 0.2122 - val_acc: 0.9215
Epoch 7/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1831 - acc: 0.9350 - val_loss: 0.1829 - val_acc: 0.9315
Epoch 8/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.1740 - acc: 0.9367 - val_loss: 0.1729 - val_acc: 0.9331
Epoch 9/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1743 - acc: 0.9398 - val_loss: 0.1658 - val_acc: 0.9385
Epoch 10/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1587 - acc: 0.9444 - val_loss: 0.1591 - val_acc: 0.9362
Epoch 11/100
5197/5197 [==============================] - 0s 15us/step - loss: 0.1532 - acc: 0.9459 - val_loss: 0.1516 - val_acc: 0.9469
Epoch 12/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1424 - acc: 0.9471 - val_loss: 0.1448 - val_acc: 0.9400
Epoch 13/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1352 - acc: 0.9500 - val_loss: 0.1319 - val_acc: 0.9500
Epoch 14/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1327 - acc: 0.9546 - val_loss: 0.1255 - val_acc: 0.9562
Epoch 15/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1274 - acc: 0.9592 - val_loss: 0.1491 - val_acc: 0.9415
Epoch 16/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1184 - acc: 0.9588 - val_loss: 0.1152 - val_acc: 0.9562
Epoch 17/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.1150 - acc: 0.9609 - val_loss: 0.1154 - val_acc: 0.9654
Epoch 18/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.1165 - acc: 0.9594 - val_loss: 0.1076 - val_acc: 0.9600
Epoch 19/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.1047 - acc: 0.9646 - val_loss: 0.1214 - val_acc: 0.9646
Epoch 20/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.1021 - acc: 0.9659 - val_loss: 0.0994 - val_acc: 0.9669
Epoch 21/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0949 - acc: 0.9692 - val_loss: 0.0995 - val_acc: 0.9731
Epoch 22/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0944 - acc: 0.9669 - val_loss: 0.0958 - val_acc: 0.9738
Epoch 23/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0954 - acc: 0.9709 - val_loss: 0.0936 - val_acc: 0.9723
Epoch 24/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0889 - acc: 0.9729 - val_loss: 0.0921 - val_acc: 0.9662
Epoch 25/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0885 - acc: 0.9706 - val_loss: 0.0881 - val_acc: 0.9731
Epoch 26/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0856 - acc: 0.9733 - val_loss: 0.0991 - val_acc: 0.9708
Epoch 27/100
5197/5197 [==============================] - 0s 15us/step - loss: 0.0837 - acc: 0.9748 - val_loss: 0.0863 - val_acc: 0.9731
Epoch 28/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0824 - acc: 0.9752 - val_loss: 0.0882 - val_acc: 0.9685
Epoch 29/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0910 - acc: 0.9729 - val_loss: 0.0824 - val_acc: 0.9754
Epoch 30/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0819 - acc: 0.9748 - val_loss: 0.0864 - val_acc: 0.9731
Epoch 31/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0813 - acc: 0.9742 - val_loss: 0.0801 - val_acc: 0.9731
Epoch 32/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0773 - acc: 0.9765 - val_loss: 0.0789 - val_acc: 0.9746
Epoch 33/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0857 - acc: 0.9734 - val_loss: 0.0881 - val_acc: 0.9692
Epoch 34/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0789 - acc: 0.9759 - val_loss: 0.0777 - val_acc: 0.9723
Epoch 35/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0814 - acc: 0.9756 - val_loss: 0.1096 - val_acc: 0.9623
Epoch 36/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0758 - acc: 0.9775 - val_loss: 0.0760 - val_acc: 0.9738
Epoch 37/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0754 - acc: 0.9767 - val_loss: 0.0770 - val_acc: 0.9738
Epoch 38/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0762 - acc: 0.9771 - val_loss: 0.0821 - val_acc: 0.9746
Epoch 39/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0865 - acc: 0.9746 - val_loss: 0.0765 - val_acc: 0.9746
Epoch 40/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0710 - acc: 0.9788 - val_loss: 0.0768 - val_acc: 0.9754
Epoch 41/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0710 - acc: 0.9802 - val_loss: 0.0752 - val_acc: 0.9762
Epoch 42/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0687 - acc: 0.9794 - val_loss: 0.0723 - val_acc: 0.9746
Epoch 43/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0697 - acc: 0.9802 - val_loss: 0.0878 - val_acc: 0.9731
Epoch 44/100
5197/5197 [==============================] - 0s 15us/step - loss: 0.0721 - acc: 0.9786 - val_loss: 0.0728 - val_acc: 0.9723
Epoch 45/100
5197/5197 [==============================] - 0s 15us/step - loss: 0.0716 - acc: 0.9792 - val_loss: 0.0714 - val_acc: 0.9723
Epoch 46/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0734 - acc: 0.9804 - val_loss: 0.0802 - val_acc: 0.9738
Epoch 47/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0749 - acc: 0.9771 - val_loss: 0.0940 - val_acc: 0.9715
Epoch 48/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0670 - acc: 0.9804 - val_loss: 0.0695 - val_acc: 0.9738
Epoch 49/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0674 - acc: 0.9802 - val_loss: 0.0714 - val_acc: 0.9769
Epoch 50/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0663 - acc: 0.9802 - val_loss: 0.0688 - val_acc: 0.9746
Epoch 51/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0662 - acc: 0.9823 - val_loss: 0.0800 - val_acc: 0.9746
Epoch 52/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0690 - acc: 0.9808 - val_loss: 0.0726 - val_acc: 0.9769
Epoch 53/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0681 - acc: 0.9811 - val_loss: 0.0913 - val_acc: 0.9692
Epoch 54/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0657 - acc: 0.9811 - val_loss: 0.0676 - val_acc: 0.9754
Epoch 55/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0694 - acc: 0.9813 - val_loss: 0.0710 - val_acc: 0.9754
Epoch 56/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0679 - acc: 0.9806 - val_loss: 0.0767 - val_acc: 0.9762
Epoch 57/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0671 - acc: 0.9806 - val_loss: 0.1178 - val_acc: 0.9623
Epoch 58/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0672 - acc: 0.9823 - val_loss: 0.0724 - val_acc: 0.9762
Epoch 59/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0651 - acc: 0.9813 - val_loss: 0.0746 - val_acc: 0.9769
Epoch 60/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0641 - acc: 0.9827 - val_loss: 0.0791 - val_acc: 0.9746
Epoch 61/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0631 - acc: 0.9827 - val_loss: 0.0676 - val_acc: 0.9785
Epoch 62/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0634 - acc: 0.9819 - val_loss: 0.0750 - val_acc: 0.9769
Epoch 63/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0689 - acc: 0.9804 - val_loss: 0.0762 - val_acc: 0.9762
Epoch 64/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0644 - acc: 0.9808 - val_loss: 0.0686 - val_acc: 0.9777
Epoch 65/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0619 - acc: 0.9823 - val_loss: 0.0648 - val_acc: 0.9762
Epoch 66/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0634 - acc: 0.9827 - val_loss: 0.0707 - val_acc: 0.9785
Epoch 67/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0707 - acc: 0.9810 - val_loss: 0.0737 - val_acc: 0.9762
Epoch 68/100
5197/5197 [==============================] - 0s 24us/step - loss: 0.0647 - acc: 0.9825 - val_loss: 0.0658 - val_acc: 0.9762
Epoch 69/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0675 - acc: 0.9802 - val_loss: 0.0695 - val_acc: 0.9777
Epoch 70/100
5197/5197 [==============================] - 0s 21us/step - loss: 0.0635 - acc: 0.9823 - val_loss: 0.0638 - val_acc: 0.9792
Epoch 71/100
5197/5197 [==============================] - 0s 21us/step - loss: 0.0628 - acc: 0.9806 - val_loss: 0.0665 - val_acc: 0.9808
Epoch 72/100
5197/5197 [==============================] - 0s 23us/step - loss: 0.0612 - acc: 0.9817 - val_loss: 0.0802 - val_acc: 0.9754
Epoch 73/100
5197/5197 [==============================] - 0s 23us/step - loss: 0.0645 - acc: 0.9810 - val_loss: 0.0684 - val_acc: 0.9777
Epoch 74/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0622 - acc: 0.9821 - val_loss: 0.0633 - val_acc: 0.9785
Epoch 75/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0623 - acc: 0.9825 - val_loss: 0.0890 - val_acc: 0.9754
Epoch 76/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0673 - acc: 0.9829 - val_loss: 0.0630 - val_acc: 0.9800
Epoch 77/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0629 - acc: 0.9811 - val_loss: 0.0733 - val_acc: 0.9746
Epoch 78/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0647 - acc: 0.9819 - val_loss: 0.0837 - val_acc: 0.9700
Epoch 79/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0622 - acc: 0.9829 - val_loss: 0.0675 - val_acc: 0.9777
Epoch 80/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0610 - acc: 0.9825 - val_loss: 0.1065 - val_acc: 0.9685
Epoch 81/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0618 - acc: 0.9815 - val_loss: 0.0626 - val_acc: 0.9808
Epoch 82/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0676 - acc: 0.9804 - val_loss: 0.0672 - val_acc: 0.9800
Epoch 83/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0609 - acc: 0.9823 - val_loss: 0.0622 - val_acc: 0.9792
Epoch 84/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0654 - acc: 0.9815 - val_loss: 0.0632 - val_acc: 0.9792
Epoch 85/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0618 - acc: 0.9815 - val_loss: 0.0653 - val_acc: 0.9785
Epoch 86/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0592 - acc: 0.9835 - val_loss: 0.0618 - val_acc: 0.9831
Epoch 87/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0600 - acc: 0.9825 - val_loss: 0.0659 - val_acc: 0.9785
Epoch 88/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0682 - acc: 0.9821 - val_loss: 0.0673 - val_acc: 0.9792
Epoch 89/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0606 - acc: 0.9829 - val_loss: 0.0764 - val_acc: 0.9754
Epoch 90/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0605 - acc: 0.9827 - val_loss: 0.0646 - val_acc: 0.9800
Epoch 91/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0583 - acc: 0.9844 - val_loss: 0.0620 - val_acc: 0.9808
Epoch 92/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0637 - acc: 0.9829 - val_loss: 0.0630 - val_acc: 0.9792
Epoch 93/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0601 - acc: 0.9827 - val_loss: 0.0682 - val_acc: 0.9777
Epoch 94/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0610 - acc: 0.9827 - val_loss: 0.0767 - val_acc: 0.9769
Epoch 95/100
5197/5197 [==============================] - 0s 19us/step - loss: 0.0623 - acc: 0.9831 - val_loss: 0.0644 - val_acc: 0.9792
Epoch 96/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0678 - acc: 0.9806 - val_loss: 0.0611 - val_acc: 0.9815
Epoch 97/100
5197/5197 [==============================] - 0s 18us/step - loss: 0.0609 - acc: 0.9815 - val_loss: 0.0780 - val_acc: 0.9762
Epoch 98/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0599 - acc: 0.9825 - val_loss: 0.0625 - val_acc: 0.9815
Epoch 99/100
5197/5197 [==============================] - 0s 17us/step - loss: 0.0746 - acc: 0.9779 - val_loss: 0.0610 - val_acc: 0.9800
Epoch 100/100
5197/5197 [==============================] - 0s 16us/step - loss: 0.0582 - acc: 0.9831 - val_loss: 0.0664 - val_acc: 0.9792
In [40]:
print("Time taken to execute model 1 ==> {} seconds".format(end_time_1 - start_time_1))
Time taken to execute model 1 ==> 9.786781072616577 seconds

A summary to show to give an overview of the model created and the number of trainable parameters among other info.

In [41]:
model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 20)                240       
_________________________________________________________________
dense_2 (Dense)              (None, 10)                210       
_________________________________________________________________
dense_3 (Dense)              (None, 2)                 22        
=================================================================
Total params: 472
Trainable params: 472
Non-trainable params: 0
_________________________________________________________________
In [42]:
T_pred = model.predict(X_test)
#Converting predictions to label
pred = list()
for i in range(len(T_pred)):
    pred.append(np.argmax(T_pred[i]))
#Converting one hot encoded test label to label
test = list()
for i in range(len(T_test)):
    test.append(np.argmax(T_test[i]))

Simple Feed Forward Neural Network Model (Regression)

The simplest kind of neural network is a single-layer perceptron network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. The sum of the products of the weights and the inputs is calculated in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically -1). Neurons with this kind of activation function are also called artificial neurons or linear threshold units. The goal of a feedforward network is to approximate some function f. For example, for a classifier, y = f(x) maps an input x to a category y. A feedforward network defines a mapping y = f(x;θ) and learns the value of the parameters θ that result in the best function approximation. These models are called feedforward because information flows through the function being evaluated from x, through the intermediate computations used to define f, and finally to the output y. There are no feedback connections in which outputs of the model are fed back into itself.

Implementation Technique

  • I will be using Keras to build the neural network model as it makes it easier to easily tweak the paramters needed to build each layer of the neural network

  • As I am building a basic neual network for my second of the 3 models to build, I will be vuilding using a 11 input layers followed by decresing number hidden layers and one output layer with 1 neurons which is the scope of the data at hand.

  • The input dimensions will be noted at 11 whihc is the attibutes that I am using as the deciding factor for the quality of wine.

  • I will not be using activation function for the output and the final activation function used for regression will be the mean squared error function.

  • Keras provides an option to compile the model using epochs and batch sizes for muliple loads to be send for compilation.

  • We will also be checking how the model performs by issuing the test data as a validation set and compiling the model.

In [43]:
X1 = X.iloc[:,0:11]
X1
Out[43]:
fixed_acidity volatile_acidity citric_acid residual_sugar chlorides free_sulfur_dioxide total_sulfur_dioxide density pH sulphates alcohol
0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4
1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.99680 3.20 0.68 9.8
2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.99700 3.26 0.65 9.8
3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.99800 3.16 0.58 9.8
4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4
... ... ... ... ... ... ... ... ... ... ... ...
4893 6.2 0.21 0.29 1.6 0.039 24.0 92.0 0.99114 3.27 0.50 11.2
4894 6.6 0.32 0.36 8.0 0.047 57.0 168.0 0.99490 3.15 0.46 9.6
4895 6.5 0.24 0.19 1.2 0.041 30.0 111.0 0.99254 2.99 0.46 9.4
4896 5.5 0.29 0.30 1.1 0.022 20.0 110.0 0.98869 3.34 0.38 12.8
4897 6.0 0.21 0.38 0.8 0.020 22.0 98.0 0.98941 3.26 0.32 11.8

6497 rows × 11 columns

In [44]:
T1 = df_all.iloc[:, 11:12]
T1
Out[44]:
quality
0 5
1 5
2 5
3 6
4 5
... ...
4893 6
4894 5
4895 6
4896 7
4897 6

6497 rows × 1 columns

In [45]:
X1_train,X1_test,T1_train,T1_test = train_test_split(X1,T1,test_size = 0.2)
In [46]:
X1_train.shape
Out[46]:
(5197, 11)
In [47]:
T1_train.shape
Out[47]:
(5197, 1)
In [48]:
model1 = Sequential()
model1.add(Dense(11, input_dim=11, activation='relu'))
model1.add(Dense(9, activation='relu'))
model1.add(Dense(7, activation='relu'))
model1.add(Dense(5, activation='relu'))
model1.add(Dense(3, activation='relu'))
model1.add(Dense(1))
In [49]:
model1.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
In [50]:
start_time_2 = time.time()
history1 = model1.fit(X1_train, T1_train, validation_data = (X1_test,T1_test), epochs=100, batch_size=32)
end_time_2 = time.time()
Train on 5197 samples, validate on 1300 samples
Epoch 1/100
5197/5197 [==============================] - 1s 151us/step - loss: 8.1728 - acc: 0.1256 - val_loss: 5.2585 - val_acc: 0.1546
Epoch 2/100
5197/5197 [==============================] - 0s 31us/step - loss: 3.1244 - acc: 0.2473 - val_loss: 0.8631 - val_acc: 0.4415
Epoch 3/100
5197/5197 [==============================] - 0s 38us/step - loss: 0.7411 - acc: 0.4745 - val_loss: 0.6403 - val_acc: 0.4969
Epoch 4/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.6316 - acc: 0.4955 - val_loss: 0.6059 - val_acc: 0.5054
Epoch 5/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.6115 - acc: 0.4947 - val_loss: 0.6026 - val_acc: 0.4723
Epoch 6/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.6065 - acc: 0.5005 - val_loss: 0.5845 - val_acc: 0.5046
Epoch 7/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5942 - acc: 0.5082 - val_loss: 0.5900 - val_acc: 0.5062
Epoch 8/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5920 - acc: 0.5107 - val_loss: 0.5729 - val_acc: 0.5031
Epoch 9/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5788 - acc: 0.5111 - val_loss: 0.5662 - val_acc: 0.5000
Epoch 10/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5718 - acc: 0.5180 - val_loss: 0.6100 - val_acc: 0.4623
Epoch 11/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5737 - acc: 0.5145 - val_loss: 0.5649 - val_acc: 0.4923
Epoch 12/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5717 - acc: 0.5224 - val_loss: 0.5677 - val_acc: 0.4885
Epoch 13/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5716 - acc: 0.5088 - val_loss: 0.5697 - val_acc: 0.4769
Epoch 14/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5681 - acc: 0.5197 - val_loss: 0.5790 - val_acc: 0.4715
Epoch 15/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5674 - acc: 0.5172 - val_loss: 0.5607 - val_acc: 0.4946
Epoch 16/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5612 - acc: 0.5199 - val_loss: 0.5549 - val_acc: 0.4938
Epoch 17/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5575 - acc: 0.5272 - val_loss: 0.5543 - val_acc: 0.4915
Epoch 18/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5574 - acc: 0.5282 - val_loss: 0.5526 - val_acc: 0.5000
Epoch 19/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5526 - acc: 0.5261 - val_loss: 0.5493 - val_acc: 0.5000
Epoch 20/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5541 - acc: 0.5136 - val_loss: 0.5466 - val_acc: 0.5031
Epoch 21/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5538 - acc: 0.5265 - val_loss: 0.5721 - val_acc: 0.5192
Epoch 22/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5560 - acc: 0.5243 - val_loss: 0.5465 - val_acc: 0.5108
Epoch 23/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5547 - acc: 0.5226 - val_loss: 0.5440 - val_acc: 0.5115
Epoch 24/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5550 - acc: 0.5265 - val_loss: 0.5495 - val_acc: 0.4969
Epoch 25/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5548 - acc: 0.5270 - val_loss: 0.5433 - val_acc: 0.5092
Epoch 26/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5511 - acc: 0.5265 - val_loss: 0.5746 - val_acc: 0.4908
Epoch 27/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5469 - acc: 0.5295 - val_loss: 0.5419 - val_acc: 0.5108
Epoch 28/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5441 - acc: 0.5266 - val_loss: 0.5378 - val_acc: 0.5100
Epoch 29/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5619 - acc: 0.5188 - val_loss: 0.5418 - val_acc: 0.5131
Epoch 30/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5445 - acc: 0.5299 - val_loss: 0.5455 - val_acc: 0.5023
Epoch 31/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5389 - acc: 0.5295 - val_loss: 0.5452 - val_acc: 0.4992
Epoch 32/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5458 - acc: 0.5224 - val_loss: 0.5441 - val_acc: 0.5131
Epoch 33/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5394 - acc: 0.5311 - val_loss: 0.5433 - val_acc: 0.5038
Epoch 34/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5458 - acc: 0.5313 - val_loss: 0.5563 - val_acc: 0.5162
Epoch 35/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5384 - acc: 0.5280 - val_loss: 0.5444 - val_acc: 0.4969
Epoch 36/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5336 - acc: 0.5318 - val_loss: 0.6306 - val_acc: 0.4923
Epoch 37/100
5197/5197 [==============================] - 0s 31us/step - loss: 0.5471 - acc: 0.5297 - val_loss: 0.5531 - val_acc: 0.5123
Epoch 38/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5385 - acc: 0.5280 - val_loss: 0.5376 - val_acc: 0.5069
Epoch 39/100
5197/5197 [==============================] - 0s 30us/step - loss: 0.5342 - acc: 0.5334 - val_loss: 0.5431 - val_acc: 0.5108
Epoch 40/100
5197/5197 [==============================] - 0s 36us/step - loss: 0.5329 - acc: 0.5305 - val_loss: 0.5451 - val_acc: 0.5115
Epoch 41/100
5197/5197 [==============================] - 0s 46us/step - loss: 0.5424 - acc: 0.5290 - val_loss: 0.5381 - val_acc: 0.5154
Epoch 42/100
5197/5197 [==============================] - 0s 32us/step - loss: 0.5335 - acc: 0.5238 - val_loss: 0.5501 - val_acc: 0.5023
Epoch 43/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5412 - acc: 0.5313 - val_loss: 0.5394 - val_acc: 0.5123
Epoch 44/100
5197/5197 [==============================] - 0s 36us/step - loss: 0.5440 - acc: 0.5315 - val_loss: 0.5735 - val_acc: 0.5131
Epoch 45/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5494 - acc: 0.5265 - val_loss: 0.5386 - val_acc: 0.5131
Epoch 46/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5384 - acc: 0.5290 - val_loss: 0.5933 - val_acc: 0.5077
Epoch 47/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5318 - acc: 0.5355 - val_loss: 0.6433 - val_acc: 0.4869
Epoch 48/100
5197/5197 [==============================] - 0s 40us/step - loss: 0.5326 - acc: 0.5372 - val_loss: 0.5356 - val_acc: 0.5138
Epoch 49/100
5197/5197 [==============================] - 0s 34us/step - loss: 0.5503 - acc: 0.5253 - val_loss: 0.5368 - val_acc: 0.5077
Epoch 50/100
5197/5197 [==============================] - 0s 38us/step - loss: 0.5264 - acc: 0.5397 - val_loss: 0.5960 - val_acc: 0.5023
Epoch 51/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5360 - acc: 0.5365 - val_loss: 0.5432 - val_acc: 0.5146
Epoch 52/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5323 - acc: 0.5338 - val_loss: 0.5407 - val_acc: 0.5085
Epoch 53/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5296 - acc: 0.5351 - val_loss: 0.5791 - val_acc: 0.4908
Epoch 54/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5369 - acc: 0.5240 - val_loss: 0.5806 - val_acc: 0.5115
Epoch 55/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5270 - acc: 0.5328 - val_loss: 0.5391 - val_acc: 0.5154
Epoch 56/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5368 - acc: 0.5465 - val_loss: 0.6203 - val_acc: 0.4654
Epoch 57/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5325 - acc: 0.5322 - val_loss: 0.5421 - val_acc: 0.5069
Epoch 58/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5326 - acc: 0.5342 - val_loss: 0.5361 - val_acc: 0.5115
Epoch 59/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5354 - acc: 0.5330 - val_loss: 0.5514 - val_acc: 0.5054
Epoch 60/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5517 - acc: 0.5232 - val_loss: 0.5497 - val_acc: 0.5215
Epoch 61/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5320 - acc: 0.5340 - val_loss: 0.5425 - val_acc: 0.5092
Epoch 62/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5263 - acc: 0.5361 - val_loss: 0.5378 - val_acc: 0.5123
Epoch 63/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5341 - acc: 0.5336 - val_loss: 0.5414 - val_acc: 0.5085
Epoch 64/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5355 - acc: 0.5382 - val_loss: 0.5530 - val_acc: 0.4985
Epoch 65/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5275 - acc: 0.5340 - val_loss: 0.5677 - val_acc: 0.5115
Epoch 66/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5343 - acc: 0.5343 - val_loss: 0.5409 - val_acc: 0.5215
Epoch 67/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5254 - acc: 0.5347 - val_loss: 0.5676 - val_acc: 0.4908
Epoch 68/100
5197/5197 [==============================] - 0s 30us/step - loss: 0.5380 - acc: 0.5234 - val_loss: 0.5540 - val_acc: 0.5315
Epoch 69/100
5197/5197 [==============================] - 0s 34us/step - loss: 0.5331 - acc: 0.5309 - val_loss: 0.5533 - val_acc: 0.5046
Epoch 70/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5293 - acc: 0.5347 - val_loss: 0.5469 - val_acc: 0.5200
Epoch 71/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5266 - acc: 0.5372 - val_loss: 0.5443 - val_acc: 0.5200
Epoch 72/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5230 - acc: 0.5390 - val_loss: 0.5503 - val_acc: 0.5285
Epoch 73/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5313 - acc: 0.5330 - val_loss: 0.6397 - val_acc: 0.4531
Epoch 74/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5331 - acc: 0.5295 - val_loss: 0.5493 - val_acc: 0.5054
Epoch 75/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5255 - acc: 0.5368 - val_loss: 0.5343 - val_acc: 0.5100
Epoch 76/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5337 - acc: 0.5299 - val_loss: 0.5413 - val_acc: 0.5254
Epoch 77/100
5197/5197 [==============================] - 0s 31us/step - loss: 0.5285 - acc: 0.5395 - val_loss: 0.5320 - val_acc: 0.5154
Epoch 78/100
5197/5197 [==============================] - 0s 41us/step - loss: 0.5294 - acc: 0.5411 - val_loss: 0.5540 - val_acc: 0.4992
Epoch 79/100
5197/5197 [==============================] - 0s 41us/step - loss: 0.5250 - acc: 0.5409 - val_loss: 0.5383 - val_acc: 0.5062
Epoch 80/100
5197/5197 [==============================] - 0s 35us/step - loss: 0.5306 - acc: 0.5355 - val_loss: 0.5316 - val_acc: 0.5185
Epoch 81/100
5197/5197 [==============================] - 0s 31us/step - loss: 0.5242 - acc: 0.5345 - val_loss: 0.5440 - val_acc: 0.5092
Epoch 82/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5378 - acc: 0.5342 - val_loss: 0.5708 - val_acc: 0.4931
Epoch 83/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5249 - acc: 0.5353 - val_loss: 0.5416 - val_acc: 0.5146
Epoch 84/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5223 - acc: 0.5413 - val_loss: 0.5327 - val_acc: 0.5162
Epoch 85/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5246 - acc: 0.5265 - val_loss: 0.5354 - val_acc: 0.5015
Epoch 86/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5271 - acc: 0.5382 - val_loss: 0.5380 - val_acc: 0.5262
Epoch 87/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5239 - acc: 0.5420 - val_loss: 0.5395 - val_acc: 0.5092
Epoch 88/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5218 - acc: 0.5392 - val_loss: 0.5575 - val_acc: 0.5262
Epoch 89/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5284 - acc: 0.5384 - val_loss: 0.5368 - val_acc: 0.5077
Epoch 90/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5279 - acc: 0.5336 - val_loss: 0.5396 - val_acc: 0.5162
Epoch 91/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5273 - acc: 0.5403 - val_loss: 0.5420 - val_acc: 0.5169
Epoch 92/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5212 - acc: 0.5399 - val_loss: 0.5331 - val_acc: 0.5131
Epoch 93/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5211 - acc: 0.5376 - val_loss: 0.5353 - val_acc: 0.5123
Epoch 94/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5290 - acc: 0.5393 - val_loss: 0.5497 - val_acc: 0.5092
Epoch 95/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5223 - acc: 0.5351 - val_loss: 0.5290 - val_acc: 0.5131
Epoch 96/100
5197/5197 [==============================] - 0s 27us/step - loss: 0.5266 - acc: 0.5438 - val_loss: 0.5353 - val_acc: 0.5062
Epoch 97/100
5197/5197 [==============================] - 0s 29us/step - loss: 0.5291 - acc: 0.5403 - val_loss: 0.5354 - val_acc: 0.5146
Epoch 98/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5208 - acc: 0.5390 - val_loss: 0.5334 - val_acc: 0.5100
Epoch 99/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5295 - acc: 0.5326 - val_loss: 0.5341 - val_acc: 0.5131
Epoch 100/100
5197/5197 [==============================] - 0s 28us/step - loss: 0.5243 - acc: 0.5349 - val_loss: 0.5300 - val_acc: 0.5215
In [51]:
print("Time taken to execute model 2 ==> {} seconds".format(end_time_2 - start_time_2))
Time taken to execute model 2 ==> 16.264949798583984 seconds

A summary to show to give an overview of the model created and the number of trainable parameters among other info.

In [52]:
model1.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_4 (Dense)              (None, 11)                132       
_________________________________________________________________
dense_5 (Dense)              (None, 9)                 108       
_________________________________________________________________
dense_6 (Dense)              (None, 7)                 70        
_________________________________________________________________
dense_7 (Dense)              (None, 5)                 40        
_________________________________________________________________
dense_8 (Dense)              (None, 3)                 18        
_________________________________________________________________
dense_9 (Dense)              (None, 1)                 4         
=================================================================
Total params: 372
Trainable params: 372
Non-trainable params: 0
_________________________________________________________________
In [53]:
T1_pred = model1.predict(X1_test)
In [54]:
T1_pred.shape
Out[54]:
(1300, 1)
In [55]:
T1_pred = np.array(T1_pred)
In [56]:
# Rounding the values to determine the exact quality of wine
T1_pred = np.round(T1_pred)

Long Short Term Memory Model (LSTM) (Classification)

Long Short-Term Memory units, or LSTMs, was proposed by the German researchers Sepp Hochreiter and Juergen Schmidhuber as a solution to the vanishing gradient problem. LSTMs help preserve the error that can be backpropagated through time and layers. By maintaining a more constant error, they allow recurrent nets to continue to learn over many time steps (over 1000), thereby opening a channel to link causes and effects remotely. This is one of the central challenges to machine learning and AI, since algorithms are frequently confronted by environments where reward signals are sparse and delayed, such as life itself. LSTMs contain information outside the normal flow of the recurrent network in a gated cell. Information can be stored in, written to, or read from a cell, much like data in a computer’s memory. The cell makes decisions about what to store, and when to allow reads, writes and erasures, via gates that open and close. Unlike the digital storage on computers, however, these gates are analog, implemented with element-wise multiplication by sigmoids, which are all in the range of 0-1. Analog has the advantage over digital of being differentiable, and therefore suitable for backpropagation.

Implementation Technique

  • I will be using Keras to build this model since it is much easier to build the model and finetune the hyperparameters

  • I will be using the LSTM layer as the primary layer where the neural network is trainied.

  • In order to ease the flow of data to the LSTM layer I'll be limiting the large input size into 256 splits.

  • The SpatialDropout will ensure that any nodes that have a probavility of 30 percent or lower will be ignored adn then sent to the LSTM

  • Applying a dropout with a 30 percent hyper parameter is sued to filter down the complex output further into a simpler output

  • Post processing will involve using the RELU activation function followed by a softmax activation application to finalise out classification model.

In [57]:
model_lstm = Sequential()
model_lstm.add(Embedding(input_dim = 6000, output_dim = 256))
model_lstm.add(SpatialDropout1D(0.3))
model_lstm.add(LSTM(256, dropout = 0.3, recurrent_dropout = 0.3))
model_lstm.add(Dense(256, activation = 'relu'))
model_lstm.add(Dropout(0.3))
model_lstm.add(Dense(2, activation = 'softmax'))
WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From C:\Users\Amit\.conda\envs\testenv_1\lib\site-packages\keras\backend\tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
In [58]:
model_lstm.compile(loss='categorical_crossentropy',optimizer='Adam',metrics=['accuracy'])
In [59]:
T2_pred = model_lstm.predict(X_test)
In [60]:
T2_pred.shape
Out[60]:
(1300, 2)
In [61]:
T2_pred
Out[61]:
array([[0.507182  , 0.49281803],
       [0.5035891 , 0.4964109 ],
       [0.5044344 , 0.49556562],
       ...,
       [0.50284517, 0.4971549 ],
       [0.5038634 , 0.49613655],
       [0.5038127 , 0.49618733]], dtype=float32)
In [62]:
#Converting predictions to label
pred2 = list()
for i in range(len(T2_pred)):
    pred2.append(np.argmax(T2_pred[i]))
#Converting one hot encoded test label to label
test2 = list()
for i in range(len(T_test)):
    test2.append(np.argmax(T_test[i]))
In [63]:
# pred2
In [64]:
# test2
In [65]:
start_time_3 = time.time()
history2 = model_lstm.fit(X_train, T_train,validation_data = (X_test,T_test), epochs=100, batch_size=64)
end_time_3 = time.time()
Train on 5197 samples, validate on 1300 samples
Epoch 1/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.4565 - acc: 0.7987 - val_loss: 0.2574 - val_acc: 0.8915
Epoch 2/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.2145 - acc: 0.9176 - val_loss: 0.1695 - val_acc: 0.9385
Epoch 3/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1612 - acc: 0.9438 - val_loss: 0.1604 - val_acc: 0.9438
Epoch 4/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1437 - acc: 0.9490 - val_loss: 0.1499 - val_acc: 0.9538
Epoch 5/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1404 - acc: 0.9534 - val_loss: 0.1538 - val_acc: 0.9431
Epoch 6/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1267 - acc: 0.9559 - val_loss: 0.1482 - val_acc: 0.9485
Epoch 7/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1217 - acc: 0.9556 - val_loss: 0.1528 - val_acc: 0.9462
Epoch 8/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1272 - acc: 0.9540 - val_loss: 0.1494 - val_acc: 0.9500
Epoch 9/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1209 - acc: 0.9548 - val_loss: 0.1525 - val_acc: 0.9438
Epoch 10/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1210 - acc: 0.9571 - val_loss: 0.1477 - val_acc: 0.9469
Epoch 11/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1131 - acc: 0.9584 - val_loss: 0.1485 - val_acc: 0.9500
Epoch 12/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1159 - acc: 0.9621 - val_loss: 0.1701 - val_acc: 0.9392
Epoch 13/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1092 - acc: 0.9619 - val_loss: 0.1533 - val_acc: 0.9500
Epoch 14/100
5197/5197 [==============================] - 5s 1000us/step - loss: 0.1115 - acc: 0.9609 - val_loss: 0.1532 - val_acc: 0.9515
Epoch 15/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.1049 - acc: 0.9602 - val_loss: 0.1535 - val_acc: 0.9554
Epoch 16/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1027 - acc: 0.9632 - val_loss: 0.1685 - val_acc: 0.9508
Epoch 17/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0983 - acc: 0.9650 - val_loss: 0.1435 - val_acc: 0.9538
Epoch 18/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.1037 - acc: 0.9607 - val_loss: 0.1448 - val_acc: 0.9554
Epoch 19/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0909 - acc: 0.9650 - val_loss: 0.1567 - val_acc: 0.9485
Epoch 20/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0903 - acc: 0.9671 - val_loss: 0.1624 - val_acc: 0.9469
Epoch 21/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0875 - acc: 0.9661 - val_loss: 0.1573 - val_acc: 0.9492
Epoch 22/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0904 - acc: 0.9652 - val_loss: 0.1548 - val_acc: 0.9515
Epoch 23/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0877 - acc: 0.9625 - val_loss: 0.1565 - val_acc: 0.9546
Epoch 24/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0765 - acc: 0.9686 - val_loss: 0.1728 - val_acc: 0.9531
Epoch 25/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0883 - acc: 0.9677 - val_loss: 0.1545 - val_acc: 0.9538
Epoch 26/100
5197/5197 [==============================] - 8s 2ms/step - loss: 0.0797 - acc: 0.9711 - val_loss: 0.1576 - val_acc: 0.9562
Epoch 27/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0755 - acc: 0.9694 - val_loss: 0.1645 - val_acc: 0.9592
Epoch 28/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0735 - acc: 0.9725 - val_loss: 0.1536 - val_acc: 0.9531
Epoch 29/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0681 - acc: 0.9750 - val_loss: 0.1609 - val_acc: 0.9492
Epoch 30/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0681 - acc: 0.9763 - val_loss: 0.1423 - val_acc: 0.9569
Epoch 31/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0708 - acc: 0.9725 - val_loss: 0.1489 - val_acc: 0.9600
Epoch 32/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0644 - acc: 0.9754 - val_loss: 0.1624 - val_acc: 0.9585
Epoch 33/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0608 - acc: 0.9761 - val_loss: 0.1622 - val_acc: 0.9554
Epoch 34/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0628 - acc: 0.9761 - val_loss: 0.1503 - val_acc: 0.9554
Epoch 35/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0610 - acc: 0.9763 - val_loss: 0.1511 - val_acc: 0.9531
Epoch 36/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0541 - acc: 0.9802 - val_loss: 0.1497 - val_acc: 0.9577
Epoch 37/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0520 - acc: 0.9792 - val_loss: 0.1758 - val_acc: 0.9538
Epoch 38/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0463 - acc: 0.9827 - val_loss: 0.2151 - val_acc: 0.9538
Epoch 39/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0445 - acc: 0.9838 - val_loss: 0.2288 - val_acc: 0.9554
Epoch 40/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0528 - acc: 0.9794 - val_loss: 0.1503 - val_acc: 0.9585
Epoch 41/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0568 - acc: 0.9781 - val_loss: 0.1860 - val_acc: 0.9600
Epoch 42/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0537 - acc: 0.9798 - val_loss: 0.1905 - val_acc: 0.9523
Epoch 43/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0403 - acc: 0.9863 - val_loss: 0.1942 - val_acc: 0.9608
Epoch 44/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0385 - acc: 0.9865 - val_loss: 0.1857 - val_acc: 0.9585
Epoch 45/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0388 - acc: 0.9863 - val_loss: 0.1773 - val_acc: 0.9623
Epoch 46/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0431 - acc: 0.9825 - val_loss: 0.1823 - val_acc: 0.9646
Epoch 47/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0368 - acc: 0.9865 - val_loss: 0.2254 - val_acc: 0.9508
Epoch 48/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0334 - acc: 0.9883 - val_loss: 0.2023 - val_acc: 0.9577
Epoch 49/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0331 - acc: 0.9881 - val_loss: 0.2123 - val_acc: 0.9638
Epoch 50/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0361 - acc: 0.9871 - val_loss: 0.1923 - val_acc: 0.9654
Epoch 51/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0341 - acc: 0.9873 - val_loss: 0.1855 - val_acc: 0.9600
Epoch 52/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0352 - acc: 0.9881 - val_loss: 0.2042 - val_acc: 0.9600
Epoch 53/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0307 - acc: 0.9871 - val_loss: 0.2026 - val_acc: 0.9600
Epoch 54/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0344 - acc: 0.9863 - val_loss: 0.1789 - val_acc: 0.9585
Epoch 55/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0296 - acc: 0.9894 - val_loss: 0.2150 - val_acc: 0.9608
Epoch 56/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0309 - acc: 0.9869 - val_loss: 0.2043 - val_acc: 0.9631
Epoch 57/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0254 - acc: 0.9910 - val_loss: 0.2108 - val_acc: 0.9623
Epoch 58/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0261 - acc: 0.9883 - val_loss: 0.2104 - val_acc: 0.9615
Epoch 59/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0183 - acc: 0.9935 - val_loss: 0.2477 - val_acc: 0.9569
Epoch 60/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0237 - acc: 0.9906 - val_loss: 0.2277 - val_acc: 0.9608
Epoch 61/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0208 - acc: 0.9925 - val_loss: 0.2674 - val_acc: 0.9608
Epoch 62/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0251 - acc: 0.9906 - val_loss: 0.2289 - val_acc: 0.9615
Epoch 63/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0215 - acc: 0.9904 - val_loss: 0.2468 - val_acc: 0.9608
Epoch 64/100
5197/5197 [==============================] - 8s 1ms/step - loss: 0.0318 - acc: 0.9896 - val_loss: 0.2088 - val_acc: 0.9577
Epoch 65/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0260 - acc: 0.9888 - val_loss: 0.2219 - val_acc: 0.9615
Epoch 66/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0208 - acc: 0.9927 - val_loss: 0.2307 - val_acc: 0.9577
Epoch 67/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0235 - acc: 0.9921 - val_loss: 0.2464 - val_acc: 0.9554
Epoch 68/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0246 - acc: 0.9923 - val_loss: 0.2207 - val_acc: 0.9592
Epoch 69/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0154 - acc: 0.9954 - val_loss: 0.2407 - val_acc: 0.9569
Epoch 70/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0163 - acc: 0.9940 - val_loss: 0.2430 - val_acc: 0.9592
Epoch 71/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0173 - acc: 0.9940 - val_loss: 0.2264 - val_acc: 0.9623
Epoch 72/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0133 - acc: 0.9948 - val_loss: 0.2413 - val_acc: 0.9654
Epoch 73/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0173 - acc: 0.9929 - val_loss: 0.2360 - val_acc: 0.9585
Epoch 74/100
5197/5197 [==============================] - ETA: 0s - loss: 0.0163 - acc: 0.993 - 6s 1ms/step - loss: 0.0165 - acc: 0.9929 - val_loss: 0.2437 - val_acc: 0.9615
Epoch 75/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0166 - acc: 0.9946 - val_loss: 0.2596 - val_acc: 0.9654
Epoch 76/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0254 - acc: 0.9917 - val_loss: 0.2454 - val_acc: 0.9646
Epoch 77/100
5197/5197 [==============================] - 8s 2ms/step - loss: 0.0106 - acc: 0.9963 - val_loss: 0.2708 - val_acc: 0.9546
Epoch 78/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0116 - acc: 0.9954 - val_loss: 0.3081 - val_acc: 0.9585
Epoch 79/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0120 - acc: 0.9948 - val_loss: 0.2692 - val_acc: 0.9577
Epoch 80/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0120 - acc: 0.9954 - val_loss: 0.2800 - val_acc: 0.9569
Epoch 81/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0167 - acc: 0.9937 - val_loss: 0.2428 - val_acc: 0.9646
Epoch 82/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0146 - acc: 0.9937 - val_loss: 0.2716 - val_acc: 0.9623
Epoch 83/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0116 - acc: 0.9962 - val_loss: 0.2861 - val_acc: 0.9638
Epoch 84/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0077 - acc: 0.9969 - val_loss: 0.3113 - val_acc: 0.9538
Epoch 85/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0140 - acc: 0.9948 - val_loss: 0.2822 - val_acc: 0.9554
Epoch 86/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0151 - acc: 0.9950 - val_loss: 0.2345 - val_acc: 0.9631
Epoch 87/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0069 - acc: 0.9981 - val_loss: 0.2748 - val_acc: 0.9646
Epoch 88/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0101 - acc: 0.9963 - val_loss: 0.2615 - val_acc: 0.9592
Epoch 89/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0139 - acc: 0.9956 - val_loss: 0.2619 - val_acc: 0.9615
Epoch 90/100
5197/5197 [==============================] - 5s 1ms/step - loss: 0.0104 - acc: 0.9969 - val_loss: 0.2838 - val_acc: 0.9615
Epoch 91/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0134 - acc: 0.9952 - val_loss: 0.2836 - val_acc: 0.9592
Epoch 92/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0067 - acc: 0.9971 - val_loss: 0.3199 - val_acc: 0.9600
Epoch 93/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0106 - acc: 0.9958 - val_loss: 0.2843 - val_acc: 0.9592
Epoch 94/100
5197/5197 [==============================] - 10s 2ms/step - loss: 0.0091 - acc: 0.9965 - val_loss: 0.2810 - val_acc: 0.9608
Epoch 95/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0077 - acc: 0.9975 - val_loss: 0.3136 - val_acc: 0.9600
Epoch 96/100
5197/5197 [==============================] - 8s 2ms/step - loss: 0.0082 - acc: 0.9969 - val_loss: 0.2664 - val_acc: 0.9654
Epoch 97/100
5197/5197 [==============================] - 6s 1ms/step - loss: 0.0091 - acc: 0.9965 - val_loss: 0.2912 - val_acc: 0.9577
Epoch 98/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0093 - acc: 0.9963 - val_loss: 0.2961 - val_acc: 0.9638
Epoch 99/100
5197/5197 [==============================] - 7s 1ms/step - loss: 0.0067 - acc: 0.9975 - val_loss: 0.2844 - val_acc: 0.9623
Epoch 100/100
5197/5197 [==============================] - 9s 2ms/step - loss: 0.0106 - acc: 0.9967 - val_loss: 0.2941 - val_acc: 0.9600
In [66]:
print("Time taken to execute model 3 ==> {} seconds".format(end_time_3 - start_time_3))
Time taken to execute model 3 ==> 639.4616613388062 seconds
In [67]:
model_lstm.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, None, 256)         1536000   
_________________________________________________________________
spatial_dropout1d_1 (Spatial (None, None, 256)         0         
_________________________________________________________________
lstm_1 (LSTM)                (None, 256)               525312    
_________________________________________________________________
dense_10 (Dense)             (None, 256)               65792     
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_11 (Dense)             (None, 2)                 514       
=================================================================
Total params: 2,127,618
Trainable params: 2,127,618
Non-trainable params: 0
_________________________________________________________________
In [68]:
T2_pred = model_lstm.predict(X_test)
In [69]:
T2_pred.shape
Out[69]:
(1300, 2)
In [70]:
T2_pred
Out[70]:
array([[3.3627977e-05, 9.9996638e-01],
       [8.9555863e-12, 1.0000000e+00],
       [3.5247927e-10, 1.0000000e+00],
       ...,
       [5.5351892e-09, 1.0000000e+00],
       [1.0000000e+00, 1.0417468e-12],
       [8.2389584e-10, 1.0000000e+00]], dtype=float32)
In [71]:
#Converting predictions to label
pred2 = list()
for i in range(len(T2_pred)):
    pred2.append(np.argmax(T2_pred[i]))
#Converting one hot encoded test label to label
test2 = list()
for i in range(len(T_test)):
    test2.append(np.argmax(T_test[i]))
In [72]:
history.history
Out[72]:
{'val_loss': [0.3108088984856239,
  0.2317090741946147,
  0.21953296546752638,
  0.20762729140428396,
  0.1987680183007167,
  0.21224595170754654,
  0.18285118075517506,
  0.1728799158334732,
  0.16575154235729805,
  0.159101204092686,
  0.15160490368421262,
  0.14477778760286478,
  0.13192366164464217,
  0.1254942344014461,
  0.14913995039004546,
  0.11521605914601912,
  0.11535687480981534,
  0.10756302628379602,
  0.12142853319644928,
  0.0994165607369863,
  0.09948620208180868,
  0.09579808811728771,
  0.0935905816921821,
  0.09212509284798916,
  0.08810986975064644,
  0.09911102578043937,
  0.08626987732373752,
  0.08819305030772319,
  0.08237071367410513,
  0.08643870670061846,
  0.08009779619482847,
  0.07888950359362823,
  0.08810364967355361,
  0.0777448422060563,
  0.10961243440325444,
  0.07600075157789084,
  0.07704449533843077,
  0.08210342777463106,
  0.07653855667664454,
  0.07681606979897389,
  0.07524085136560293,
  0.07228094439093884,
  0.08775430846672792,
  0.0727609961021405,
  0.0714292240028198,
  0.08022205204344712,
  0.09398945349340256,
  0.06949784469719117,
  0.07137042581461943,
  0.0688160701841116,
  0.07998167003576571,
  0.0725590397302921,
  0.09131851010597669,
  0.06757812595138182,
  0.07100005680551896,
  0.07666762423629944,
  0.11782133508187075,
  0.07240035644230934,
  0.07461346578426085,
  0.07909530970912713,
  0.06756520914343687,
  0.07499931617425039,
  0.07619905307745704,
  0.06855545662343503,
  0.0648062521018661,
  0.07069278738819636,
  0.0737100966503987,
  0.06575625895880736,
  0.06947715642647101,
  0.06382278565890515,
  0.06650677909071628,
  0.08016534722768344,
  0.06842659262367166,
  0.06327968309131952,
  0.08899735274796303,
  0.06301618600980594,
  0.07332864172183551,
  0.08365588337182998,
  0.06753117551597265,
  0.10647837729551471,
  0.06260930279699656,
  0.06724754352982228,
  0.062217944929232966,
  0.06317663110792637,
  0.06525783861199251,
  0.06175793560365072,
  0.06592878794727416,
  0.0672680932870851,
  0.07638146173495512,
  0.06458284781242793,
  0.06196015474887995,
  0.06301817447233658,
  0.06822212143299672,
  0.07668567016577492,
  0.06444842233107641,
  0.06110286489415627,
  0.07803084364304176,
  0.06249029024002644,
  0.061000554297979064,
  0.06642801503722484],
 'val_acc': [0.8769230769230769,
  0.9161538457870484,
  0.9184615380947406,
  0.9230769227101253,
  0.9284615382781396,
  0.9215384611716637,
  0.9315384611716637,
  0.9330769227101252,
  0.9384615384615385,
  0.9361538459704473,
  0.946923076923077,
  0.94,
  0.95,
  0.9561538461538461,
  0.9415384615384615,
  0.9561538461538461,
  0.9653846153846154,
  0.96,
  0.9646153846153847,
  0.9669230769230769,
  0.9730769230769231,
  0.9738461538461538,
  0.9723076923076923,
  0.9661538461538461,
  0.9730769230769231,
  0.9707692307692307,
  0.9730769230769231,
  0.9684615384615385,
  0.9753846153846154,
  0.9730769230769231,
  0.9730769230769231,
  0.9746153846153847,
  0.9692307692307692,
  0.9723076923076923,
  0.9623076923076923,
  0.9738461538461538,
  0.9738461538461538,
  0.9746153846153847,
  0.9746153846153847,
  0.9753846153846154,
  0.9761538461538461,
  0.9746153846153847,
  0.9730769230769231,
  0.9723076923076923,
  0.9723076923076923,
  0.9738461538461538,
  0.9715384615384616,
  0.9738461538461538,
  0.9769230769230769,
  0.9746153846153847,
  0.9746153846153847,
  0.9769230769230769,
  0.9692307692307692,
  0.9753846153846154,
  0.9753846153846154,
  0.9761538461538461,
  0.9623076921242933,
  0.9761538461538461,
  0.9769230769230769,
  0.9746153846153847,
  0.9784615384615385,
  0.9769230769230769,
  0.9761538461538461,
  0.9776923076923076,
  0.9761538461538461,
  0.9784615384615385,
  0.9761538461538461,
  0.9761538461538461,
  0.9776923076923076,
  0.9792307692307692,
  0.9807692307692307,
  0.9753846153846154,
  0.9776923076923076,
  0.9784615384615385,
  0.9753846153846154,
  0.98,
  0.9746153846153847,
  0.9699999998166011,
  0.9776923076923076,
  0.9684615384615385,
  0.9807692307692307,
  0.98,
  0.9792307692307692,
  0.9792307692307692,
  0.9784615384615385,
  0.9830769230769231,
  0.9784615384615385,
  0.9792307692307692,
  0.9753846152012164,
  0.98,
  0.9807692307692307,
  0.9792307692307692,
  0.9776923076923076,
  0.9769230769230769,
  0.9792307692307692,
  0.9815384615384616,
  0.9761538461538461,
  0.9815384615384616,
  0.98,
  0.9792307692307692],
 'loss': [8.43063307274115,
  0.2653856799422215,
  0.23007536353641964,
  0.21763757140746087,
  0.20267141502144842,
  0.19964391287012653,
  0.18311713582160313,
  0.173954164331547,
  0.17425164568732743,
  0.15873429764901556,
  0.1531598614842382,
  0.1423638250498857,
  0.13515328692603115,
  0.13271854375878825,
  0.12735199827356064,
  0.11836538193806538,
  0.11503119309957334,
  0.11648549829291246,
  0.10468417011833613,
  0.10208275553562307,
  0.09490833868309698,
  0.09444113525749973,
  0.09539705960480369,
  0.0888544456653006,
  0.08851670112682347,
  0.08560356133837344,
  0.0836717191268578,
  0.08235270954196858,
  0.09104642507971217,
  0.08186921052819654,
  0.0812716029128513,
  0.0772599957851818,
  0.0857319906282269,
  0.07885130523218888,
  0.0814355338032049,
  0.07583560585568495,
  0.07541375850682674,
  0.07619937318983802,
  0.08645768877662172,
  0.07099737257033455,
  0.070982461470135,
  0.06870885322356805,
  0.06974332822726532,
  0.07210656968715078,
  0.0716284139697222,
  0.0733716533285825,
  0.07490400325965957,
  0.06704431478624416,
  0.06739870848573927,
  0.06631858722887946,
  0.06619171258323762,
  0.06895853100892545,
  0.06813011704307004,
  0.06573161686064018,
  0.06941943117147693,
  0.06787422593207755,
  0.0670853302962268,
  0.06721721484309877,
  0.06508423149237531,
  0.06409448438928131,
  0.06314909735173761,
  0.0633822935067973,
  0.0688977822883971,
  0.06439836050584634,
  0.06191528665232663,
  0.06337555561215402,
  0.07069640843370098,
  0.06471938690952334,
  0.06754302546490906,
  0.06345340653097019,
  0.0627638761578488,
  0.061196772942168264,
  0.06451371762050684,
  0.06222660690746744,
  0.062270233496143336,
  0.06734066359831024,
  0.06289756706219947,
  0.06465013708441887,
  0.06215893928508518,
  0.06102742351318211,
  0.06179371080777956,
  0.06762945869704932,
  0.06085622101471657,
  0.06540998038194273,
  0.06175254306374076,
  0.059222538535634965,
  0.059956769214649874,
  0.06820715193765563,
  0.06058440661591499,
  0.06052138212919866,
  0.05827573238774684,
  0.06369555148168644,
  0.06009255014381455,
  0.06104941690784706,
  0.06234540172818706,
  0.06782570034542836,
  0.06091396048290869,
  0.05988785726686143,
  0.07456480496187944,
  0.05821902973963385],
 'acc': [0.4098518376215527,
  0.9009043679389674,
  0.9141812584297873,
  0.921685587839138,
  0.9270733115373494,
  0.9291899172714267,
  0.934962478364365,
  0.9366942466922464,
  0.9397729459303444,
  0.944390994804695,
  0.9459303444294785,
  0.9470848566480662,
  0.9499711372060043,
  0.9545891860803549,
  0.9592072349547055,
  0.9588223975370406,
  0.9609390032711179,
  0.9593996536463344,
  0.9645949586299788,
  0.9659418895516644,
  0.9692130075157984,
  0.9669039830786231,
  0.9709447758436799,
  0.9728689628631904,
  0.970559938437484,
  0.9732538002693862,
  0.9747931498941698,
  0.9751779873118347,
  0.9728689628861285,
  0.9747931498941698,
  0.9742158937848759,
  0.9765249182220512,
  0.9734462189839532,
  0.9759476621242265,
  0.9755628247065615,
  0.9774870117375409,
  0.9767173369366182,
  0.977102174331345,
  0.9746007311910718,
  0.9788339426592265,
  0.9801808735809121,
  0.9794111987685203,
  0.9801808735809121,
  0.9786415239675976,
  0.9792187800654224,
  0.98037329228401,
  0.977102174331345,
  0.9803732922954791,
  0.9801808735809121,
  0.9801808735809121,
  0.9822974793264585,
  0.9807581296902059,
  0.9811429670964018,
  0.9811429670964018,
  0.9813353857994997,
  0.980565710987108,
  0.980565710987108,
  0.9822974793149895,
  0.9813353857994997,
  0.9826823167211853,
  0.9826823167211853,
  0.9819126419087936,
  0.98037329228401,
  0.980758129701675,
  0.9822974793379275,
  0.9826823167211853,
  0.9809505483933039,
  0.9824898980180874,
  0.9801808735809121,
  0.9822974793149895,
  0.980565710987108,
  0.9817202232056956,
  0.9809505483933039,
  0.9821050606233606,
  0.9824898980295564,
  0.9828747354242833,
  0.9811429670964018,
  0.9819126419087936,
  0.9828747354242833,
  0.9824898980180874,
  0.9815278045255358,
  0.98037329228401,
  0.9822974793149895,
  0.9815278045025977,
  0.9815278045025977,
  0.9834519915335771,
  0.9824898980180874,
  0.9821050606118915,
  0.9828747354242833,
  0.9826823167211853,
  0.9844140850490668,
  0.9828747354357523,
  0.9826823167211853,
  0.9826823167326544,
  0.9830671541273812,
  0.980565710987108,
  0.9815278045025977,
  0.9824898980180874,
  0.9778718491437368,
  0.9830671541273812]}

IV - Results

  • Presents the results of applications of your deep networks.
  • Visualize the results
  • Discuss about the choice of network structures and performance of it as you change the structures.
  • What do you think about the results?

Simple Feed Forward Neural Network Model (Classification)

Applying simple feed forward networks to the classification model yeilded a very good result since the softmax fucntion at the last layer provides a very good activation function for classfication. We can see the accuracy for the model oscillate between 97 and 99 percent.

In [73]:
# IF GPU
# plt.plot(history.history['accuracy'])
# plt.plot(history.history['val_accuracy'])
# IF CPU
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

The loss for the testing model is seen to be very less compared to the training showing that the mode does better as the number of epochs keep rising. Whilke there are ocassinal spikes in the graph indicating the model has not performed that well onm an average we can see a stable low loss compared to the traning set.

In [74]:
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

The accuracy is calculated for the predictions made and we can see that the model created is highly accurate

In [75]:
a = accuracy_score(pred,test)
print('Accuracy is:', a*100)
Accuracy is: 97.92307692307692

Simple Feed Forward Neural Network Model (Regression)

Applying simple feed forward networks to the regression model yeilded a very average result since the least square error function at the last layer didn't prove to be the best that the model could do with a regression problem. We can see the accuracy for the model oscillate between 49 and 57 percent wihch is not a good indicator of this model's performance.

In [76]:
# IF GPU
# plt.plot(history1.history['accuracy'])
# plt.plot(history1.history['val_accuracy'])
# IF CPU
plt.plot(history1.history['acc'])
plt.plot(history1.history['val_acc'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

As we saw from the previous graph it is a given that with a low accuracy the loss would be much higher than expected. We can clearly see that there is no overlap in the train and testing loss functions which impllies that the chances of hvaing a successful prediction are very low.

In [77]:
plt.plot(history1.history['loss'])
plt.plot(history1.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

The accuracy score for the regression model turned out to be very low due ot the low accuracy and high loss seen in the graphs above

In [78]:
b = accuracy_score(T1_test, T1_pred)
print('Accuracy is:', b*100)
Accuracy is: 52.15384615384615

Long Short Term Memory Model (LSTM) (Classification)

LSTM proved to be a little more challenging to apply on the data which focused more on tuning the data. From the graph below we can see that although the training has shown good accuracy the test data doesn't seem to converge. This leaves us with a lower accuracy score in the testing data. But nonetheless the model shows good promise with a high testing accuracy oscillating between 95 and 97 percent.

In [79]:
# IF GPU
# plt.plot(history2.history['accuracy'])
# plt.plot(history2.history['val_accuracy'])
# IF CPU
plt.plot(history2.history['acc'])
plt.plot(history2.history['val_acc'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

Of all the mdoels tested soi far, LSTM has shown the lowest stable loss function values on the test set. While it is certainly higher on the test set compared to the training set which can be seen in the accuracy score below, this nevertheless showcases that the predicts made by this model are good.

In [80]:
plt.plot(history2.history['loss'])
plt.plot(history2.history['val_loss'])
plt.title('Model Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper right')
plt.show()

The high accuracy and low loss form the last two grpahs show a very good accuracy score second only to the simple feed forward model used for classification

In [81]:
c = accuracy_score(test2, pred2)
print('Accuracy is:', c*100)
Accuracy is: 96.0

Conclusions

Discuss the challenges or somethat that you learned. If you have any suggestion about the assignment, you can write about it.

This has been one of the more interesting challenges to tackle since there is a lot of plug and play to get a highly accurate score on the predictions. This has not only taught me how to design neural models but also taught me how minute modifications to the network and sometimes a complete network overhaul can give a much better score on the predictions.

One of the challenges in this assignment was getting the underlying libraries to run on GPU's directly since I was testing the code directly on my personal; device since I have a CUDA supported graphics driver. The testing speed has been completely different with GPU's racing through the models much faster than the CPU. Setting up the GPU instance of tensor flow was the real challenge since there were a lot off dependencies on the CUDA SDK and its libraries and I chose Windows as my platform of choice since it was better supported.

References

[1] Keras Documentation, https://keras.io/

[2] Long Short Term Memory Model, TowardsDataScience.com, https://towardsdatascience.com/machine-learning-recurrent-neural-networks-and-long-short-term-memory-lstm-python-keras-example-86001ceaaebc

[3] Regression Using Keras, MachineLearningMastery.com, https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/

[4] Long Shoret Term Memory Model, SkyMind.ai, https://skymind.ai/wiki/lstm

[5] TesnorFlow GPU documentation, TesnorFlow.org, https://www.tensorflow.org/install/gpu

Grading

Only well-written notebook will be graded. Please follow the structure and fill in as the other assignments.

extra credit points description
1 First structure (implementation, explanation, plot results, discussion of results)
1 Second structure (implementation, explanation, plot results, discussion of results)
1 Third structure (implementation, explanation, plot results, discussion of results)
1 Explaining and discussing the reason for the selection (Any relation to your data?)
Comparing the results, discuss or verify your choice

Optional Extra Credits

  • [OPT 1] Test your program with GPUs and compare training speed. You can try FloydHub for free 2 hours of GPU resources or MAMBA cluster for this test.
  • [OPT 2] Implement one of neural networks in the chart above and apply it to some real data.

Code for OPT 1 Extra credit

The commands below are to showcase the hardware used in testing the models which include

  • the CPU info
  • the GPU info and
  • the device used by tensorflow for processing
In [82]:
!wmic cpu get name
Name                                      

Intel(R) Core(TM) i5-9300H CPU @ 2.40GHz  



In [83]:
!nvidia-smi
Fri Dec 06 14:07:08 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 431.94       Driver Version: 431.94       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0  On |                  N/A |
| N/A   50C    P5    24W /  N/A |   1065MiB /  6144MiB |      9%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       760    C+G   ...rosoft Office\root\Office16\WINWORD.EXE N/A      |
|    0       788    C+G   ...entUtility\HPSystemEventUtilityHost.exe N/A      |
|    0       872    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
|    0      1344    C+G   Insufficient Permissions                   N/A      |
|    0      4460    C+G   ....14003.0_x64__8wekyb3d8bbwe\GameBar.exe N/A      |
|    0      4528    C+G   ...ogram Files\Mozilla Firefox\firefox.exe N/A      |
|    0      4652    C+G   ...x64__8wekyb3d8bbwe\Microsoft.Photos.exe N/A      |
|    0      5040    C+G   C:\Windows\explorer.exe                    N/A      |
|    0      5904    C+G   ...9.0_x64__8wekyb3d8bbwe\WinStore.App.exe N/A      |
|    0      7432    C+G   ...ogram Files\Mozilla Firefox\firefox.exe N/A      |
|    0      8160    C+G   ...0.0.0_x64__8wekyb3d8bbwe\Calculator.exe N/A      |
|    0      8544    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0      8940    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0      9720    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     10272    C+G   ...1.85.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     11212    C+G   ...54.91.0_x64__kzf8qxf38zg5c\SkypeApp.exe N/A      |
|    0     11996    C+G   ...6\win32\OmenCommandCenterBackground.exe N/A      |
|    0     12396    C+G   ...common\wallpaper_engine\wallpaper32.exe N/A      |
|    0     12916    C+G   C:\Program Files\LGHUB\lghub.exe           N/A      |
|    0     13068    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0     13772    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     15248    C+G   ...ogram Files\Mozilla Firefox\firefox.exe N/A      |
|    0     15696    C+G   ...ogram Files\Mozilla Firefox\firefox.exe N/A      |
|    0     15784    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
+-----------------------------------------------------------------------------+

This is used ot check which device Keras will use to perform the neural network processing.

In [84]:
# If the output here is blank if it is running on CPU

import tensorflow as tf
sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(log_device_placement=True))
Device mapping:

Time Comparison between CPU and GPU

As part of the extra credit I will be using the CPU and GPU shown above and calculate the time it takes to execute the models

Running on CPU

  1. Simple Feedforward Model (Classification) --> 11.90 seconds with 98.38% accuracy

  2. Simple Feedforward Model (Regression) --> 18.20 seconds with 49.69% accuracy

  3. Long Short Term Memory (Classification) --> 604.39 seconds with 96.54% accuracy

Running on GPU

  1. Simple Feedforward Model (Classification) --> 30.94 seconds with 97.76% accuracy

  2. Simple Feedforward Model (Regression) --> 74.79 seconds with 56.46% accuracy

  3. Long Short Term Memory (Classification) --> 139.56 seconds with 94.69% accuracy

Observations

We have come across a really interesting observation while performing our comparison

  • Simple feedforward networks which should normally run fast on GPU's seem to be running faster on GPUs with an almost equal accuracy
  • The biggest benefactor is the LSTM model where we see more than a three fold increase in the time it takes to execute the whole model. This shows how complex neural networks can take advantage of the higher power of the GPU's processing core